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ABSTRACT 

Evaluations of multisite programs present special 
challenges to evaluators concerned with the use of evaluation 
results. This paper offers an approach that holds promise for 
mitigating the problems inherent in evaluating programs at multiple 
sites. A brief analysis is presented of the challenges these programs 
present and the reasons these challenges exist. In sum, these 
challenges exist because each site implements a program differently. 
Two federal education programs, the Educational Partnerships Program 
and the Star Schools distance learning program, are described and are 
used as examples throughout the paper. A second section describes a 
general theoretical approach to evaluation that is based on the 
theory-driven evaluation approach of H.-T. Chen (1989) and the work 
of Jacques Derrida (1981) and others who emphasize the deconstruction 
of concepts and activities. The evaluation framework, illustrated in 
figure 1, incorporates the elements of: (1) context; (2) problem 
definition; (3) program; (A) funded projects; (5) activities; and (6) 
outcomes. Addressing program developers' values and assumptions is 
particularly useful in multisite program evaluations. (Contains 42 
references.) (SLD) 
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INTRODUCTION 

Evaluations of multisite programs present special challenges to evaluators concerned with the use 
of evaluation results (Tushnet, 1993). This paper offers an approach thai holds promise for 
mitigating the problems inherent in evaluating programs implemented in a number of different 
sites. The Introduction of this paper includes a brief analysis of the challenges presented by 
multisite program evaluations and the reasons these challenges exist In addition, the Introduction 
briefly describes two federal education programs, the Educational Partnerships Program and the 
Star Schools distance learning program, because they will be used as examples throughout the 
p^er. The second section of the paper describes the general theoretical approach to evaluation. 

Challenges of Multisite Evaluations 

Most federal programs, and frequently state and foundation-supported programs as well, are 
implemented at several sites. Program intentions differ, depending on the nature of the program, 
with some federal programs designed to be similar across sites and others in which variation is 
both expected and desired. For example. Chapter 1 of the Elementary and Secondary Education 
Act provides educational support for educationally disadvantaged children across the nation. 
Although different treatments are expected depending on the characteristics of students and 
communities, presumably there are fairly large similarities across sites. In contrast, demonstration 
programs, such as the Educational Parmerships Program described in this paper, are designed to 
have maximum variability. In the middle are programs, such as Star Schools, that fund several 
projects expected to differ from one another, however, activities within sites served by a single 
project are expected to be similar. 

Whatever the design of the program, since the 1970s, researchers and evaluators have noted 
that program implementation differs at each site (Berman & McLaughlin, 1976). Goggin, 
Bowman, Lester, and O'Toole (1990) note that even for laws, implementation varies according to 
context. Variations are even greater for programs. This variation across sites leads to one 
challenge to evaluators because it tnay be difficult to untangle program effects from effects 
generated by the conditions at a single site, including site biases about program participants 
(Sinacore & Turpin, 199 1 ). 

A second challenge comes from the reasons implementations vary by site. Some variation 
may result from a local project's assessment of the availability of appropriate staff. Local leaders 
may, for example, change the configuration of project services because individuals with the 
requisite background and experience are unavailable, as when a program that includes 
psychological support for underachieving students uses support groups rather than individual 
counseling because trained social workers or psychologists are not available in the community. As 
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Mowbray and Herman (1991) put it, "How does one assess the effect that variability in staff 
composition has on client outcomes?" (p. 49). 

Similarly, some variation in local implementation may result from differences in how 
personnel at different sites assess client needs. Within Qiapter 1, for example, two types of 
treatments are allowed by regulation. One focuses on individual students, providing remediation 
and other supports for students identified as not achieving as well as their age mates. The other 
focuses on the school. These schoolwide Chapter 1 projects are allowed when the number of 
disadvantaged children is great enough that local educators believe that across-the-school changes 
are needed to provide adequate^educational opportunities. 

Some variation is difficult to predict and hard to measure. For example, in the Star Schools 
Program, students taking the same course transmitted by satellite to multiple sites had very 
different experiences. The national evaluation revealed that the differences were not due to 
"objective" characteristics of the students, the schools, the studio teacher, the content of the course, 
or even the background and education of the local facilitator. Less tangible matters, such as the 
interest of the facilitator and school and district leaders, seemed to accoimt for the differences 
(Tushnet et al., 1994). 

Perhaps most important, individuals at different sites interpret programs differendy. 
According to Goggin et al. (1990), "Interpretation is a function of context...a single message, such 
as a federal statute, may be interi)reted differently in different states" (p. 33). Fullan (with 
Steigelbauer, 1991) notes that educational innovation involves continuous, and developing, 
understanding of the substance of the change by school personnel. As a result, program 
implementation is an ongoing process, during which the nature of the innovation as well as current 
practice may change (Berman & McLaughlin, 1977). 

Multisite evaluations also sire challenging because they most frequently rely on some amount 
of decentralized data collection (Sinacore & Turpin, 1991). To the extent that program developers 
allow local sites to determine the data to be collected, there wiU be variation in what information is 
available to the cross-site program evaluator. Further, in both the Educational Partnerships and 
Star Schools programs, local site.s were found to collect litde information about students or other 
service recipients. In particular, little infomiation was available related to demographics and 
outcomes. Even when local sites collect data, problems of quality control arise (Sinacore & 
Turpin, 1991). 

In sum, multisite evaluations are challenging because each site implements a program 
somewhat differently. As a result, what is assumed to be a single progiam c(Hnprises, in reality, 
multiple treatments that may be impossible to describe in advance. Consequently, program 
evaluators should be as interested in describing the program, in its many manifestations, as in 
assessing effects. As will be argued later in this paper, descriptions of programs, projects, and 
activities are data for deconstructing assumptions and values and the basis for the usefulness of the 
evaluation. 
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Descriptions of the Programs 



Among the programs being evaluated by the Policy and Evaluation Center of the SouAwest 
Regional Laboratory (SWRI^) ate two national, multisite programs. The first is a demonstration 
program, the Educational Partnerships Program, designed to provide information about the role of 
partnerships that involve public schools, businesses, social service agencies, and cultural 
institutions in schools and their support in the community. The second is the Star Schools 
Program, a federally supported distance education program designed to improve opportunities for 
students in remote rural and educationally disadvantaged urban schools. Each of these piograms is 
described in turn. 

The Educational Partnerships Program 

Congress enacted the Educational Partnerships Aa in 1988 (title VI, subtitle A, Chapter 5 of the 
Omnibus Trade and Competitiveness Act of 1988, Pub. L. 100-418). The intent of tiie legislation 
was to stimulate the creation of educational partnerships in order to demonstrate their contributions 
to educational reform. The authorizing legislation requires documenting the partnerships that 
receive assistance, assessing their impact on educational institu/ons, evaluating the extent to which 
they improve their commuruties* climate for support of education, and identifying promising 
activities. The legislation goes beyond the simple evaluation question of whether funded projects 
meet their own objectives. It questions program impact on the participating institutions and their 
communities. The broader question is whether partnerships (and Uie particular activities tliey 
sponsor) can be a force to renew education and encourage community support 

The Educational Partoerships Program is administered by the Educational Networks 
Division, Programs for the Improvement of Practice, Office of Educational Research and 
Improvement (OERI), U.S. Department of Education. OERI funded four cycles of partnership 
projects for a total of 29 projects. The projects differ in structure and objectives. They include 
partnerships designed to facilitate the transition from school to woric; improve instruction in 
mathematics and science; provide opportunities for gifted students, those at risk of school failure, 
and noncoUege bound students; and stimulate systemic reform. The partnerships include a variety 
of configurations of businesses and industries, cultural institutions, health and human service 
agencies, institutions of higher education, state education agencies, and elementary and secondary 
schools. 

The documentation and evaluation study is being conducted during a three-year period. Tlie 
study was designed to identify partnership structures and activities associated with successful 
implementation, positive impact, and long term institutionalization. In addition to sponsoring a 
program evaluation, OERI required each funded project to implement a local evaluation. SWRL 



was charged with providing technical assistance to the local evaluators. OERI believed that ihc 
technical assistance would result in higher quality local evaluations. 



The Star Schools Program 

The federal government, through what is known as the Star Schools Program Assistance Act (20 
U.S.C. 4081), has encouraged: 

Improved instruction in mathematics, science, foreign languages, and other 
subjects, such as literacy skiUs and vocational education...to serve underserved 
populations, including ddsadvantaged, illiterate, limited-English proficient, and 
disabled students tiirough distance learning technologies. CPL 102-103 Aug. 17, 
1991) 



The Star Schools Program supports a variety of services, including: 

• developing or acquiring programming in various curriculum areas; 

• supporting teacher ttaining and staff development to improve instruction in 
matiiematics, science, and foreign languages; 

• providing tutorial services for students by using a variety of technologies; 

• maintaining testing services for courses offered; and 

• supporting the acquisition of telecommunication facilities and equipment, both 
by projects and by participating schools. 

The program evaluation focuses on a series of questions posed in the authorization of Star 
Schools. The questions address issues related to project organization and impact on students, 
staff, and schools. Congress expressed interest in: (a) whether courses generated through Star 
Schools last beyond federal funding, (b) the extent to which Star Schools increased access to 
courses for students, and (c) the effectiveness of Star Schools courses and staff development 
opportunities. 
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THEORETICAL APPROACH 



The framework for creating useful multisite evaluations draws greatly from two sources. First, it 
reflects some root concerns addressed by Chen (1989) in his work on "theory-driven evaluation,** 
although it lacks the heavy quantitative framework. The second source is ttit work of Denida 
(1981) and others who direct attention to "deconstruction" of concepts and activities, although it 
lacks the heavy political ideology generally associated with the deconstructionists. In this section, 
I discuss the contributions of each as well as the reasons their ideas are promising for increasing 
the usefiihiess of evaluations. Examples from the two programs are interwoven with the 
theoretical grounding. The theoretical discussions will be followed by an explanation of each 
element of the overall evaluation approach as well as descriptions of appropriate methods to use in 
explicating and deconstructing program assumptions and values. 



Theory-Driven Evaluation 

Chen and his colleagues argue that an overriding purpose of evaluation is to interpret the effects of 
various treatments on the populations of concern. Doing so, they argue, requires going beyond the 
"black box*' approach G-ipscy, 1993) that simply looks at outcomes to understanding what 
happened to produce the results, at least in nonexperimental designs. Because it is difGcult and 
generally undesirable to create true experiments for social programs, alternative approaches to 
understanding program dynamics are needed. Proponents of theory-driven evaluation maintain 
that "treatment theory** should form the basis of the evaluation. Lipsey (1993) lists the following 
elements of adequate treaunent theory: 

(1) a problem definition that specifies what condition is treatable, for which 
populations, and under what circumstances...;(2) speciflcation of the critical inputs, 
...the interrelationships anoong the inputs, and the basis forjudging magnitude or 
strength of inputs...; (3) the important steps, links, phases, or parameters of the 
transformaticm process that the treatment brings about, the intervening or mediating 
variables on which the process is contingent, and the crucial interactions with 
individual diflferences, timing, mode of dtelivery, or oiher relevant circumstances; 
and (4) specification of the expected output...and the interrelationships or 
contingencies among the outputs, (p. 1 1) 

The evaluation is driven by the: 

theory or model of how a program is supposed to work. The theory of tiie 
program's operation is then used as a basis for designing an evaluation of the 
program. By explicitly relating program processes to outcomes, this approach 
seeks to avoid the limitations of the methods-driven and black-box approaches to 
evaluation. (Smith, 1993, p. 3) 



The treatment theory, then, provides tiie evaluator with the framework that governs tiie 
evaluation. It not only generates appropriate questions about outcomes, but also duects attention to 



questions about the processes that lead to the outcomes. Evaluators can seek indicatois about the 
populations receiving treatment, the inputs, the various necessary linkages, and outcomes. 

Frequently, as in the case of the Educational Partnerships Program, program developers have 
only implicit theories about how the program is supposed to work. One desired outcome was 
increased community support for education. The logical chain from the formation of parmerships 
to increased community support was never spelled out, but rather was taken as a matter of faith. 
As evaluators, we could look for indications of greater community support (e.g., passage of bond 
and tax levies, positive newspaper reports, responses to interview questions), but interpretation 
was difficult unless we could construct a logic (a "treatment theory") of how parmerships engender 
such positive outcomes. The logical line was as follows: 

In many conununities, fewer adults have children attending public school than in 
earlier times. Consequently, adult views about schools are based on impressions 
gathered from newspaper articles about youth and casual interaction with young 
people in such places as shopping malls. Neither of these sources leads to a 
positive image for schools. Educational parmerships, on the other hand, provide an 
oppormnity for adults to get to know young people as well as teachers and other 
school personnel. Such loiowledge will create understanding for tiie chsdlenges 
faced by schools and the honest, hardworking efforts school people make to meet 
those challenges. As a result, participants in educational partno^hips will become 
advocates for schools among their fnends, thus increasing support for education. 

The usefulness of presenting this logical construction to program developers was manifested 
in two ways. First, on the federal level, the developers were able to judge whether particular 
projects' approaches to a parmership had a plausible relationship to this outcome. According to 
Chen and Rossi (1983), a lot of approaches are implausible. In this light, it is not surprising that 
few of the parmerships had an impact on community support because most involved few 
conmiunity members. It is unrealistic to expect tiiat such a goal will be met simply as a result of an 
educational parmership. Federal personnel as well as project staff members began to look at how 
to use the parmerships to increase community awareness rather than rely on their existence to do 
so. This led to some altered project designs. 

The second reason for making the program theory explicit relates to the multisite nature of the 
evaluations. By raising to consciousness the implicit theories held at both the federal and project 
sites, differences could be noted. This was most evident in the Evaluation of the Star Schools 
Program. The Star Schools legislation and program announcements focused on three distinct 
goals: increasing access to courses; demonstrating educationally sound uses of technology; and 
supporting reform, particularly in schools serving disadvantaged children, through distance 
learning. For federal policymakers and program developers, the three goals were envisioned as 
mutually supportive, but funded projects tended to. emphasize one goal. Each project had an 
implicit, or more rarely, explicit, program theoiy guiding its activities. For exan^le, projects that 
focused on providing courses to students, mainly in mral areas, to which they would otherwise not 
have access built few support mechanisms for the fecilitators at the receiving site, identified 



teachers with excellent qualifications to serve as studio teachers, and todk full responsibility for the 
quality of the curriculum and instruction. The logic was: 



The problem is that some students don't have the opportunity to take some courses 
because dieir schools are too small to offer them or &ere are too few smdents 
interested to warrant hiring a teacher. All that is needed is a good teacher in the 
studio teaching a course that other students receive. At die site, the facUitator will 
simply make sure students follow the curriculum, do their homework, and take 
their tests. Student learning of fairly traditional subject matter is the expected effect 
of distance learning. 

In contrast, the projects that emphasized distance learning as an agent of educational refoam 
had the following theory: 

Schooling as cunentiy experienced does not meet the needs of learners or society. 
Major changes am needed to support a new paradigm of education, and distance 
learning can provide impetus for those changes. Studio teachers teach in a new 
'"constructivist" manner, both to provide students with high quality opportunities 
for learning and to model cutting edge pedagogy for receiving site teachers. The 
studio teacher and the classroom teacher are a team, and effects should be on both 
students and teachers. 



The situation resulted in virtually all project staff members feeling misunderstood and 
devalued. They believed that they were always being compared to ihe other projects and always 
found lacking. Each believed that the standards by which they were judged were designed for a 
different type of project As a result, it was difficult for federal program personnel to create the 
collaboration they desired anoong grantees. The evaluation report's analysis of these underlying, 
different, program assumptions allowed projects to begin more productive conversations. 
(However, all projects believed the report underplayed the importance of their particular approach!) 

The examples given so far are tiiose in which die program theory was implicit In fact, 
implicit tiieories were more common in botii the Star Schools.and Educatk)nal Parmerships 
programs. However, some projects, particularly parmerships concerned with the transition firom 
school to wojdk, had explicitly stated and developed program theories. In such cases, the woik of 
the evaiuator was easier because the challenge was limited to developing appropriate indicators for 
the elements in the theory rather than first explicating the fhcory and then developrng the indicators. 
A further complication arose for project staff, both in the partnerships and Star Schools programs, 
who revised and refined their theories — and actions — ^as a result of learning from the evaluation or 
experience. While such strategic adjustments were more likely to lead to positive outcomes than . 
maintaining a theory and its related activities (Tushnet et al., 1993), die changes also require 
continuous adjustments by the evaluators. Program staff use of evaluation results seems to lead to 
harder, altiiough more satisfying, work for die evaiuator. 
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Deconstruction 

The contribution of deconstructivisin to multisite program evaluation lies in its focus on 
"problematizing" that which is taken foe granted. Deconstiiictionists approach texts in literature or 
data in research with an "all-encompassing critical skepticism about knowledge" (Vidich & Lyman, 
1994, p. 41). Such skepticism frames evaluation questions of how programs should operate and 
who they should benefit It also directs the evaluator to look for hidden messages embedded in Ae 
program and its implementation, particularly those that reinforce social and economic inequalities. 
The evaluator deconstructs in order to gain knowledge of the real social justice issues and 
outcomes of a program. The evaluator also deconstructs her or his own influences on design, data 
collection, and interpretation (Derrida, 1976) and sometimes offers alternative perspectives on the 
same piece of evidence. 

Although different in emphasis, deconstructivism relates closely to constructivist 
epistemologies (Schwandt, 1994). Both arc concerned with how individuals interpret the meaning 
of their position and actions, and both focus on meaning as constracted through social interaction 
(Berger & Luckmann, 1966). Because meaning is constructed, according to this argument, it 
reflects commonly accepted relationships and has a tendency to support the stams quo. It is the 
task of ^^construction, then, to raise questions about relationships and ideas — to make them 
problematic. Of prime importance to deconstractionists are power relationships. For example, in 
looking at the large numbers of female teachers and small munbers of female administrators, the 
deconstructionist asks questions about the nature of power and gender in schools and society. 
Even more radically, deconstmctionists "expose the ideological and political meanings that circulate 
within the text, particularly those which hide or displace racial, class, ethnic and gender biases" 
(Denadn, 1992, p. 151). For evaluators, the issue is not so much "text" as "program." Or as Flax 
(1987) puts it, deconstructivism seeks "to distance us from and tiiake us skeptical about beliefs 
concerning truth, knowledge, power, the self and language that are often taken from granted within 
and serve as legitimation for Western culture" (p. 624). 

Among the issues deconstractionism raises for evaluation is the question of "how voices of 
participants are to be heard, with what authority, and in what form" (Olesen, 1994, p. 167). 
Olesen argues that the question of voice is closely related to "the account," that is, the narrative that 
frames the data. She also notes that typical interviews and questionnaires, even open-ended 
interviews, of "participants" are frequently insufficient tt) give voice to those who have been 
voiceless. The questions frame the issues — help construct the reality — as does the age, gender, 
ethnicity, and class of the interviewer. 

Clearly, the deconstructionist project has political implications, but deconstmctionists argue 
that all research has (disguised) political implications. One job of the evaluator, then, is to probe 
for tile political meanings of programs. For multisite programs, the political meanings are 
constructed at the program (national) level and witiiin each site. Further, the deconstructive 
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program creates the appropriate skepticism about responses to questions, and, indeed, the 
questions themselves. 

An example of how a deconstructive cast of mind can advance an evaluation is drawn from 
the Star Schools Program. As noted above, the program had three rather different types of goals, 
and each project seemed to chose one for focus, although no formal decisions were made to tiiat 
effect One common thread ran across all program goals, and that was concern for educationally 
disadvantaged children and youth, with authorization and appropriation tied to Chapter 1. Indeed, 
all projects involved students who were disadvantaged, either by reason of socioeconomic 
conditions or living in rural, isolated areas. However, the data indicated great differences in the 
types of programs offered tt> youth in schools serving low ectxiomic areas and those in rural 
schools: 

The data related to student characteristics in schools served indicate differences 
between schools receiving fuU courses and those receiving supplementary 
instmction. Students enrolled in advanced cotirses and foreign languages are more 
likely to be white and less likely to be poor than are students enrolled in 
supplemental courses. When such data are reported within a single school, 
observers ftcquently note that tracking is occurring. (Tushnet et al., 1994, p. 97) 

In this example, the evaluators made definitions-in-use of "educationally disadvantaged" 
problematic. Rather than merely reporting different types of disadvantage addressed by the 
distance learning program, we looked at the specific characteristics by type of course. In this case, 
we found that the multiple goals of the program coupled with each project's different kind of 
attention to educationally disadvantaged students had an unintended twist 

Deconstructing program theories, particularly when they are implicit theories, is central to the 
evaluation approach. As the example indicates, even within a single program, local implementers 
frequently differ in then: understanding of the problem and what will solve it The following 
contrasts the theories held by one project that provided students with access to courses, mainly in 
rural areas, and another that embraced supplementary instruction in the cause of educational 
reform: 

1 . Students, mainly in rural areas but occasionally in urban centers, are deprived 
of equal educational opportunity because the schools tiiey attend are not able to 
offer particular courses, mainly in science and foreign languages. 
Consequently, offering such courses through distance learning will mitigate 
their disadvantage. 

2. In urban centers, students do not have the same educational opportunities as 
their suburban peers because urban teachers are less likely to have participated 
in st{^ develq>ment on new approaches to teaching mathematics. Further, 
because students enter school with less home and community experience in 
mathematics, they need supplementary instruction if they arc to achieve what 
they are capable of achieving. Both problems lead to offering supplemental 
instruction in which the studio teacher nnodels appropriate behavior. 
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The deconstructionist notes the difference in defining "disadvantage," and notes as well that 
the first ^preach is likely to reinforce current distributions of power by increasing the number of 
middle class whites who are eligible for elite institutions. Whether the second {^proach supports 
or changes social relations requires additional information, but it has the potential for making 
changes. 

Peiiiaps the most significant example of how deconstruction assisted in a program evaluation 
comes from the study of the Educational Partnerships Program. OERI funded six partnerships that 
included a focus on the transition from school to work. Most also focused on preparing studrats 
for a more technically sophisticated workplace than their parents experienced. Further, local 
program developers, community members, and business representatives expressed the belief (or 
hope) that if they developed a pool of highly trained workers, new industries would locate in the 
community. The "deconstractive" questions were: 

1. Who participates in the training, by race, gender, and socioeconomic 
characteristics? 

2. Do young people who participate in school-to-work programs get high tech 
jobs? Are there differences by race, gender, and socioeconomic group of those 
who get jobs? 

3 . Do new industries locate in the community? If so, did the existence of the labor 
pool influence the decision? Do locals get the "good" jobs? 

The data are not yet in regarding either attraction of new industries oc placements in high tech 
jobs. Students who participated in the school-to-work programs were more likely to get jobs than 
their classmates who did not, but the jobs tended to be part time, without benefits — characteristics 
of most new jobs across the nation. Two projects bucked this trend, both in the South. 

One project leads to further questions about the distribution of benefits £com the programs. 
That project, serving a prototypical rust belt city and its surrounding county, has many 
components. However, a major thrust is a school-to-work transition program, which includes a 
career education component for elementary and secondary schools. It also includes hi^ tech- 
oriented training in conjunction with a major manufacturer at the regional vocational school that 
primarily serves students from the city. A third element in the transition component is training for 
careers related to robotics located at the suburban high school farthest fixmi the city. Overall, there 
is a balance in the number of students served in the city and the county. A similar overall balance 
exists in the demographic characteristics of students. However, to the extent that data are 
available, it looks as if the most challenging, high tech programs have greater enrollment from the 
more middle class parts of the city and the suburbs. 

The final example of deconstruction in evaluation also comes from the Educational 
Partnerships Program. In virtually every funded site, during the evaluation visit, someone would 
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say, "You know, it takes a whole village to raise a child." Deconstructing flie village metaphor 
became important to the evaluation (Tushnet, 1994). In many instances, for example, the village 
was one that was — ^and should be — ^run by a small group of "elders," mainly white, mainly male. 
Their views on "raising a child" included helping students understand their place, creating "good 
workers," and, in general, perpetuating their privilege. In other instances, far fewer in number, 
the village metaphor was used in a way that called forth images of town meetings with broad 
participation and stimulating debate. 

We pursued the metaphor by asking all participants, including those who received services 
from the partnerships, what meaning they took from its use. By doing so, we were able to 
develop multiple program narratives — giving voice to those frequently overlooked in studies. We 
found interesting contrasts in interpretations of the project, some of which were rooted in different 
unders' andings of the metaphor. Some projects, including most schooi-to-woik partnerships, 
would have communicated more clearly, and experienced fewer problems, if they had relied on 
business-oriented metaphors, what we came to call the "language of the deal." 

Participants in programs being evaluated are not always comfortable with deconstructive 
activities. From their perspective, evaluation is designed to help them manage the program, and 
raising questions about social impacts can be seen as inappropriate. In addition, deconstruction 
includes a view of social justice that is not necessarily shared by project staff. The view is rooted 
in Rawls' (1971) analysis of justice as fairness, and its role in evaluation has been well explicated 
by House (1993). The deconstructionist evaluator must take care to deconstruct his or her own 
questions and conclusions and provide opportunities for alternative views and interpretations to be 
included in reports. 

The Evaluation Framework 

The framework for evaluation proposed in this paper is based on the theoretical stances described 
above. This section includes a discussion of the elements of the framework itself. Throughout, I 
indicate appropriate methods and how each element contributes to the usefulness of the approach. 
Figure 1 illustrates the evaluation framework for multisite program evaluation designs. 
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Figure 1 

Frameworkfor the Development ofMultisite Program Evaluation Designs 




Context 

Context, at the federal, state, and project site levels, frames what occurs during program 
implementation and has a major effect on program outcomes. On the federal and state levels, the 
concerns that led to the identification of the problem and other, related programs are key contextual 
matters. For project sites, context includes concerns, other activities, the history of innovation, 
demographics, and the characteristics of students, the community, and the professional staff of 
schools. Contextual variables relevant to an evaluation are identified by reference to the 
appropriate literature related to the content of the program and, particularly for demonstration 
programs, literature related to innovation and change. Consequently, other contextual factors also 
may be important for particular programs. For example, in the Star Schools Program, the 
existence or lack of an infrastructure to support technology was a key contextual feature 
influencing program implementation and outcomes. This section includes a discussion of sources 
of information about federal, state, and project site context 
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Federal and State Context 



At both the federal and state levels, key contextual information relates to the franoing of particular 
policies. Studies of policy formation point to a number of key factors in development of new 
policies and their related programs. Pressures for change come from a number of sources, 
including social movements, issue networics (Hcclo, 1978), and perceived crises (Polsby, 1984). 
According to Mazzoni (1991), these combine to create the context for policy change and, according 
to McDonnell & Eknore (1987), are combined with knowledge and assumptions about what types 
of actions would address the problem to determine the appropriate policy instrument Further, as 
Wirt, Mitchell, and Marshall (1988) argue, how the concems are responded to is largely ftamed by 
the state and federal "culture." State cultures differ, and federal cultures may chan^ overtime, but 
the key issue is the values by which decisions are made. For example, Wirt, Mitchell, and 
Marshall maintain that three values — choice, efificiency, and equity — ^frame much state education 
policy, and states balance them in different ways. In some states, and at times in the federal 
govemment, equity issues dominate. In others, and at other times at the federal level, choice is 
most important. 

In addition to gathering information about the concems that led to the development of the 
program, the evaluator gathers information about other concems in the nation and states. This 
provides a broad framework for identifying the policy environment. Further, the evaluator also 
collects information about other, related activities being implemented. 

The methods used to synthesize contextual information involve content analysis and 
interviews. Content analysis can be applied to various types of data, altiiough it is most commonly 
thought of in relationship to documentary analysis. It allows information about phenomena of 
interest to be placed in standard categories, no matter what language is used. Content analysis 
enables quantification of textual data (Patton, 1980) that can be related to quantitative findings. 
The typology for coding information from policy and program documents is guided by preliminary 
understanding of the purpose and structure of the program under study (Hall, Hord, & Griffin, 
1980; Rossman & Wilson, 1985). As analysis proceeds, new categories of interest are generated 
and findings from various data sources woven together (Schermerhom, WiUiams, & Dickison, 
1982). In complex, multisite program evaluations, content analysis is applied to the documents 
generating die program, including testimony, debate, and the Congressional Record. 

Interviews arc used to provide a basis for understanding the reasons the program took the 
form it did. The basic question is: "Why did you choose that approach?" (As the following 
sections reveal, interviews of program developers address context, definition of the problem, and 
program design.) 

Context changes over time and may affect die design, implementation, and outcomes of the 
program. For example, the Star Schools Program always mcluded goals related to integrating 
distance learning with educational refonn. However, at die inception of the program, educational 



reform was cast in terms of "more" — ^morc advanced placement courses, more hours, more 
graduation requirements. This approach was reflected in the program's emphasis on full courses 
for high school students. Later, the educational reform was defined as "standards-based systemic 
reform," and Star Schools encompassed supplementary instruction that modeled the type of 
constructivist learning promulgated by the various educational standards projects. Evaluation 
reports that noted that changes in the program focus reflected chants in the policy context were 
enlightening to both federal program and local project staff The substance of contextual change, 
particularly in the policy arena, is a prime candidate for deconstruction because such changes 
frequently reveal changes in the values held by key decisionmakers. 

Project Site 

Evaluators are interested in concerns and activities at the project sites for tiie same reasons they are 
interested in federal and state concerns. That is, the conce^ms frame the program. Programs that 
address, either direcdy or indirecdy, a number of highly salient issues are more likely to be 
successful than ones that are seen as tangential to the core issues at the site. For exan^le, 
educational partnerships that focused on the transition from school to work were more likely to be 
successful in communities in which economic development was deemed important than were 
parmerships that focused on the gifted and talented. 

Demographic data also are important, particularly for interpreting the impact of the program. 
Such data generally include information about students (gender, ethnicity, socioeconomic status), 
teachers (gender, ethnicity, years of experience), and the community. Data about student 
attendance, achievement, violence and vandalism, drug use, and language proficiency also are 
candidates for inclusion. Other variables also may be of interest, e.g., the number of computers in 
the schools or the history or interorganizational arran^ments, depending on the focus of the 
program. 

Evaluators use existing information, drawn from local, state, and federal databases, to 
develop the necessary information about the project site. However, there are limits to the amount 
of desired data available because states and districts frequendy do not have sophisticated 
information systems. Further, few projects engage in the type of needs assessment mat would 
build their information base. Ctonsequentiy, evaluators gather as much relevant information as is 
available and paint a general picture of the context. If particular missing data take on importance 
during the course of the study, the evaluator must decide whether to alter the planned approach to 
data collection in order to gather additional contextual information. In the Star Schools Program, 
for example, the projects did not collect information about ethnic or socioeconomic characteristics 
of schools participating in the program. Without such data, it was impossible to tell if the program 
fulfilled its goal of equalizing educational opportunity. Consequentiy, evaluaton sought such 
information. 
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The site context influences implementation and outeomes (Fullan, with Steigelbauer, 1991). 
Equally important, such data provide the basis forjudging how equitable program effects are 
distributed. 

Problem 

Most programs result fiom policymakers' and program developers' identification that a problem 
exists. Frequently, the problem has been an ongoing one, but is "noticed" in the policy arena as a 
result of key events. For example, public attention to sexual harassment, a phenomenon 
experienced by many women, was drawn as a result of the U.S. Supreme Court appointment of a 
man accused of woriqplace harassment Similarly, most educational programs are developed in 
response to persistent problems in education that are brought to public consciousness at particular 
times. 

Problems exist, but it is the definition of the nature of a problem that leads to the development 
of specific programmatic responses. The program theory and deconstructive approaches, then, are 
applied to how policymaters and program developers define the problem. This includes 
examination of the evidence used to establish the existence of the issue and justify the particular 
approach to addressing it embodied in the program. In the Star Schools Program, for example, 
information about the lack of advanced courses in rural areas was brought to bear as the program 
developed. The data were used politically by a Southern senator whose state was involved in 
major educational reform and who wanted federal support for the efforts, and substantively, in 
defining the curriculum subjects that were to be addressed. 

The evaluator is equally concerned about relationships program developers see between the 
problem they are addressing and other problems. In the Star Schools Program, a concern for 
fostering educational use of technology was coupled with a general understanding that schools 
serving educationally disadvantaged children were likely to require assistance in purchasing 
equipment, which then could be used to support then- learning. Perhaps more important, the 
assumed relationships are the basis for the program theory. Lipsey (1993) argues that program 
theory specifies the problem, including independent variables (generally, the treatment), and the 
nature of the subjects who are supposed to benefit fiom the treatment Because program 
developers are seldom explicit about their theories, the evaluator begins by "discovering" the 
program theory. 

The deconstmctionist approach also aids understanding how program developers view 
relationships among problems. From that perspective, the evaluator probes for program 
developers' assumptions (or lack thereof) of how program will be distributed among participants. 
By deconstructing such presumed effects as well as each element in the theoiy, the evaluator gains 
understanding of intended, although not necessarily acknowledged, effects of the program on 
social relationships. For example, the Educational Partnerships Program generally reinforced 
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existing social relationships, as indicated by the eximq>les of potential partners included in program 
documents. It included business and industry, institutions of higher education, social service 
agencies, and cultural institutions. The list lacked mention of uni<ms, communiQr-based 
organizations, or religious institutions — organizations that fi%quently have woridng and poor 
people in policymaking positions. 

Evaluators use two methods to examine the evidence that a problem exists, how it is de^ed, 
and how it relates to other problems and social and economic inequalities: content analysis and 
interviews. Content analysis is implemented on program documents, grants announcecoents, 
guidelines to reviewers, and other relevant materials. Interviews are important for gaining 
understanding of the social meanings program developers bring to their task and how they interpret 
the problem. The next section describes the impoitance of interviews in descrilnng the program 
theories, and the assumptions that frame program development. 

Program 

Procedurally, evaluators give attention to the design of the program at the same time as they are 
seeking understanding of how program developers define the problem to which they are attending. 
The most important method at this stage is the use of interviews. Interviews have a special place in 
constructing and deconstructing program theories. They provide the opportunity for the evaluat(»: 
and program developer to develop a shared understanding of the program. Through interviews, 
program developers reveal the social meanings (Denzin, 1994) they ascribe to program elements 
and are able to engage in critical or self-reflection (Kincheloe & McLaren, 1994). Although the 
evaluator enters the interview with a general protocol of questions to ask drawn from the content 
analysis, the purpose is to probe for the connections made, or not, by the program developer. It is 
frequendy useful to pose hypothetical problems, or what I have told interviewees are "mischievous 
questions," to gain access to implicit theories. For example, from one perspective, educational 
partnerships can be viewed as extra or antidemocratic. That is, they give some decisionmaking 
authority over schooling to a group that has not been elected to make such decisions. Federal 
program developers at first seemed surprised at questions cast in this light but were able to 
explicate why they thought partnerships actually enhance democratic institutions through the 
building of community. Implicit theories, encompassing assumptions, and values thus became 
explicit 

The process of surfacing the assumptions and values that lay behind the program is what 
most program developers find useful. It provides an opportunity for them to reflect on what diey 
have created, and federal policymakers, whether congressional staff or executive branch 
bureaucrats, believe they have litde time in the ordinary course of business for such reflection. The 
process also raises their consciousness about contradictions in the program. They are then able to 
decide whether they wish to live with the contradictions in the interest of enhancing pluralism or 



alter the program. For example, the Star Schools Program contains objectives related to three 
distinct goals: providing courses for students who would not otherwise have access to them; 
assisting in educational reform; and demonstrating educational {^plications of technology. 
Although these are not mutually exclusive goals, their existence has led to some confusion and 
much negative competition among projects. When federal program developers were asked about 
the three goals, they reflected a bit and acknowledged the differences among them for the first time. 
This enabled the program staff to communicate with projects \a ways that decreased rancorous 
competition and helped build collaboration anxmg the ^ded projects. Disagreements between 
staff at diff^nt projects remained, but the disagreements could be accepted more easily when 
projects did not fear that federal monitors prcfmed the objectives pursued by another project 

Funded Projects 

The same questions about assumptions are raised with regard to funded projects, which can be 
seen as the operationalization of program developers' aspirations. The attention to program 
theories and values held at each implementing site is a distinguishing marie of the {^proach to 
evaluation embraced here. That is, this approach recognizes that the same process of creating 
understanding of the problem and figuring out the activities that are likely to make things better 
occurs at each site as at die federal level. Differences among sites are nof a matter of "policy driff ' 
or "implementation failure" (Pressman & Wildavsky, 1973) but reflect the very nature of multisite 
programs. In part, differences in program theories reflect differences in the interpretation of die 
federal program. Goggin et al. (1990) say of federal legislation: 

Interpretation is a function of context..a single message...may be interpreted 
differentiy in different states. Even within a single state, the pluralistic and 
interactive nature of local-level policy adoption makes die probability of multiple, 
conflicting messages high." (p. 33) 

The differences also reflect analysis of local conditions (context), including the form the 
problem takes and the knowledge, skills, and interests of those who wiU implement the program 
(Mowbray & Herman, 1991). Perhaps naost important, however, the program theory at the federal 
level — the assumptions about relationships between elements of the "treatment" and how they will 
address the problem — ^is generally implicit. Consequently, project developers bring their own set 
of assumptions and values to beai' on project design. In multisite programs, then, we can expect 
multiple program theories. 

Evaluators address the same issues and use the same methods when they gather infotmation 
from the funded projects as they do in addressing program design. They seek project staff 
members' interpretation of the problem, as revealed in the proposal and other relevant documents. 
As well, they seek to explicate the theory and values held by project developers about how their 
program addresses the problem in their particular context 
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The information thus gained is useful in three ways. Rrst, it is useful to the evaluators 
because it provides a framework for understanding implementation and outcomes. As such, it 
gives a map of whrare to look for activities and impacts, peihaps different from those indicated by 
federal policymakers. When tiiwe arc differences between the guidance provided by the local 
project tiieory and die overarching program theory, evaluators arc advised to seek information 
about both. 

Second, analysis of the theories and values embedded in local project designs is illuminating 
to local project personnel in the same way(s) as the program analysis. Interviews serve to spur 
reflection. And, the summary analysis is shared with staff as a dialogue. Not all project managers 
greet deconstructive interpretations of their work v/ith applause, but if the ground rules are that 
tiiey are allowed to enter alternative interpretations, then most find the process helpful to program 
development at the site level. Indeed, the alternative interpretations also are indications of the 
theory and values that guide program implementation and also may help explain effects. 

Third, the set of project analyses are useful to federal program developers. TTiey can alter 
guidelines for projects if there are too many differences in theories or too few theories that address 
what the program developers think are die major issues. For example, the Educational 
Partnerships Program began witii broad guidelines for eligible projects. Program managers 
assumed tiiat most applicants for funding would be broad-based parmerships tiiat addressed poUcy 
as well as program. The assumption was based on then: reading of the literature and their 
disseminating research syntheses about partnerships to all potential applicants. In fact, however, 
most applicants had a small group of partners and focused on a fairly nanow change. After the 
first year of the evaluation, when OERI received project descriptions that included analyses of the 
local assumptions about the role of partnerships in school improvement, federal staff rewrote the 
grant guidelines to encourage more partnerships focused on systemic change. The guidelines 
became more explicit about tire theory held by federal developers about how educational 
parmerships could contribute to educational reform. 

Activities 

Program and project assumptions are played out tfirough the activities implemented by the project 
The activities operationaliz/i tire assumptions and values of project-level staff. It is sometimes 
useful to assess fidelity of implementation (Berman & McLaughlin, 1977), but it is generally more 
fruitful to study tire activities that are implemented. The basic question is, "Who does what to 
whom and witii what quality?" 

The quality dimension addresses tiie extent to which project activities reflect what is known 
about die appropriate approaches to addressing the problem. Of course, definitions of quality 
reflect values, so tfie definitions are subject to deconstniction. For example, much current 
educational reform is designed to accomplish "standards-based systemic reform," and program 
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quality is judged by the extent to which activities supportive of such leform are implemented. But 
what are the "standards"? What social meanings do they embody? Apple (1990) provides a model 
of how deconstruction of standards can be done. 

It is important to assess implementation for two reasons. First, evaluations that simply focus 
on outcomes may be evaluating "non-events" (Charters & Jones, 1973). Second, and more 
important, most new approaches comprise "innovation configurations" (Heck, Steigelbauer, Hall, 
& Loucks, 1981), with multiple activities making up the whole treatment As a result of different 
program theories, each site is likely to configure the program somewhat differcntiy. When the 
evaluation is completed, then, global statements about program impact are useless without an equal 
understanding of the relationship of particular configurations of activities to outcomes. In the 
Educational Parmerships Program, for example, a summary statement as to whether partnwships 
help support positive change would notprovide information to those who wish to improve their 
partnerships or others seeking to start a new one. It is much more useful to conclude with 
statements like, "Educational partnerships ihat include a^^z activities were more likely to have a 
positive impact on students than were those diat did not. Further, community impact was gained 
when partners include abc agencies or individuals." 

The methods for documenting and evaluating implementation include content analysis of 
documents/noting, for example, the number of workshops offered and the number of individuals 
attending. Interviews also are important sources of implementation data. Evaluators also can 
observe workshops, classes, or meetings of governing boards, using either qualitative protocols or 
quantifiable observation instruments. Surveys of project participants oi' recipients of services also 
are useful. 

From the perspective espoused in this paper, data collection concerned with implementation 
must be fairly fine grained. It must attend to differential implementation for different participants. 
For example, there is much data to indicate that in classrooms, girls and boys are treated differendy 
(Sadker & Sadker, 1994). Consequentiy, classroom observations of a "new" curriculum should 
include attention to gender differences during implementation. The deconstmction of 
implementation in the Star Schools Program led to the conclusion that white, rural, 
socioeconomically secure students were receivmg advanced courses and minority, urban, poor 
students were receiving supplementary instruction. This understanding, then, was useful when 
results were interpreted. 

The perspective on evaluation embraced in this paper includes data gathering activities ftom 
participants and recipients of program services. It is important that then* voices be included in the 
study, although for students, particularly, issues of privacy must be dealt with. Despite the 
difficulties, evaluators should attend to those who are fi«quentiy vdceless. One educational 
partnership, for example, was a dropout "reclamation" alternative school, which provided 
education to students who had been out of school for over sue months. Many were runaways or 
living on their own. The school provided self-esteem classes, computer-assisted individualized 
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instruction, workplace preparation, and opportunities to meet employers. Set in a mall, the school 
was small and had a low student-teacher ratio. The theory behind the program was explicit: The 
types of youth served by the program had low self-esteem, which stemmed from their many 
failures, and needed to develop self-discipline. When interviewed, students talked about the 
teachers* "compassion.** For most, the school was a counseling center, and they internalized the 
values central to the self-esteem class, which were that they were responsible for their own actions 
and that tiieir choices determined tiieir success. Few were employed, and they ascribed their 
unemployment totally to their "choices" but were disappointed that school staff did not use their 
contacts to set up job interviews for them. Some were even angry. Both the positive and negative 
feelings speak to issues of implementation and quality. 

The voice of service recipients can be heard through their actions as well as through 
interviews, although interviews are probably always useful in gaining understanding of the reasons 
for actions. In one Educational Parmerships project, for example, attendance at staff development 
sessions continuously decreased until, in the second year of the project, literally no one showed 
up. We found that the teachers did not see the connection of the staff development to the in-class 
activities and were quite articulate about its lack of relevance. Project staff were faced with a series 
of choices, including doing a better job of explaining the relationship; improving the relationship; 
or accepting that the in-class activities did not require staff development support Their choice of 
the third alternative helped refine our understanding of their assumptions and values and supported 
a different interpretation of program outcomes from the first year's explanation. 

Implementation evaluation is frequently cast as "process evaluation" and seen as formative. 
Indeed, evaluating the in^lementation of a program has an important role in formative evaluation. 
However, it also has a role in "impact evaluation" because it provides information necessary for 
detennining if the treatment has merit (Scriven, 1991). From both vantage points, evaluations tiiat 
include information about the implementation and quality of activities are judged useful by program 
and project developers. One function the evaluator can fiU is to establish systems for monitoring 
implementation that can be used by local project staff so they can adjust activities as needed. 

Outcomes 

Tb' final category of concern is outcomes, which includes impact, including how effects are 
distributed among various groups, and institutionalization. Impact includes both the desired 
impacts of the program and projects and the unintended consequences. Institutionalization can be 
seen as the "stabilization of use of the innovation" (Hubeiman & Miles, 1986, p. 11) or "what*s 
left" after external funding is no longer available. It involves the stabilization and continuation of 
practices and support for Uie program, with ongoing adjustments to changing context 
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Impact 



The overarching impact question is: "Did the program 'fix' the problem it was designed to 
address?" Data about the extent to which it did can be drawn froin a variety of sources, including 
surveys, student testing, interviews, and observations. The evaluators' decisions about 
appropriate measures are related to the desired outcomes. For example, it would be wrong to use 
standardized tests to measure the extent to which students engage in constructivist thinking. 
However, it might be appropriate to use standardized tests to see whether students taught in 
constructivist ways could meet "old" as well as "new" standards. 

Frcquendy, evaluators seek data only ftiom those upon whom the program is designed to 
have an impact If a program is designed to improve student performance, for example, it is 
conmion to seek outcome data only from students. I argue that such information is not sufficient; it 
is necessary to fmd out the interpretations teachers or others give to the nature of the outcomes. 
Their views are important because they have ongoing contact with students. If they believe 
improvement in student performance is important and actually occurring, their interactions with 
students are likely to change. However, if they dismiss the change as trivial or doubt its existence, 
little change will occur. 

Two examples drawn from a single school in an Educational Partnerships project illustrate 
the point The substantive focus of the program was using local area networks (LANs) installed in 
participating inner-city schools as an impetus to changing teachers' instructional practices. The 
goal was to change from teacher-centered to "student-centered, inquiry-based" insuiiction. After 
two years in the program, student outcomes were strong on performance assessments. A first- 
grade teacher embraced the technology because she saw what it could do. She said, "I keep 
hearing about what Hrst graders can do and not do. But now I know that first graders can do what 
first graders can do." Her students were engaged in demanding activities that required 
sophisticated critical thinking. In contrast, a fourth-grade teacher reading the same results said, 
"These performance tests don't count When these kids go for the kind of jobs they'll be ready 
for, no one's going to care whether they can think." And her students spent most of their day 
doing typical "drill and skill" activities, interrupted occasionally by the use of the LAN. 



Institutionalization 

Funders are particularly interested in finding out what happens when external support stops. One 
of the congressionally mandated questions asked about the Star Schools Program, for example, 
was, "How effective are teleconmiunications partnerships and programs after federal funding 
ceases?" Similarly, in the Educational Partnerships Program, OERI was interested in whether the 
activities espoused by die partnerships lasted after federal funding ceased and also whether the 
partnerships remained in existence. Unfortunately, funding foe evaluations generally ceases at the 
same time as funding for the program so it is difficult to gather information about 
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institutionalization. However, indicators do exist, including whether funded positrons axe shifted 
to nonproject budgets and whether provisions are made to orient new staff members so diey, too, 
can implement the program. 

Data for evaluating the level of institutionalization come from content analysis of documents, 
including budgets, courses of studies, plans for staff development, and contracts for equipment 
maintenance. We found in the Star Schools Program that enrollment in courses declined each year 
without subsidies from the federal grant, and by the fourth year after federal funding ceased, they 
no longer existed. 

Interviews with participants also are an important source of information about 
institutionalization, particularly when the study ends at the same time as the project The questions 
focus on plans being made and values placed on the activities. 



24 



CONCLUSION 



I have argued that evaluations that address program developers' values and assunq>tions are 
particularly useful in multisite program evaluations. SWRL used this general approach in two 
national evaluations, and the results of the studies have been used by both project and program 
personnel. However, the analysis of the approach presented here is likely to raise questions about 
whether it is realistic. Can any evaluation support aU the activities delineated? In this concluding 
section, I address that concem. 

Clearly, the approach requires evaluators with great skill and knowledge. They must have 
the aH)ropriate technical training to: design valid interview protocols, observational schemes, and 
survey instruments; develop sampling plans; conduct content analysis; and analyze and interpret 
data. More important, they must have substantive knowledge of the program focus in ordCT to 
recognize assumed relationships that are part of the program theories espoused by program and 
project developers. And most difficult, they must have a broad and deep imderstanding of tibe 
society in order to engage in the deconstnictive tasks. 

Does such a person exist? Probably not, particularly because evaluators generally work in a 
number of content areas. Even those of us who confine ourselves to education evaluate a variety 
of different substantive programs, including those designed to improve curriculum (and in diffa:ent 
substantive areas), school organization, and the achievement of particular groups of students. 
However, it is possible to gather teams of people who can share their knowledge while holding 
particular responsibility for one aspect of the evaluation. 

The second practical barrier to implementing the evaluation approach presented here lies in 
the amount of resources it requires. As outlined above, the evaluation approach is labor intensive 
and requires multiple types of data from multiple sources. Seldom are sufficient funds available. 
However, the approach offered in this paper is a model, and I do not expect any agency, including 
SWRL, will ever fulfill all its requirements. 

The approach has shown itself to be a useful model in our wotk. It enables us to create 
meaningful fiamewoiks for each evaluation we conduct We are aware of what we lose when we 
decide not to focus on a particular part of the model, and can open convo-sations with clients about 
the tradeoffs involved in our design. Perhaps most important, the approach is a constant reminder 
to probe beneath the surface to the social meanings of program elements and effects. It was natural 
for SWRL evaluations to deconstruct the different meanings of the village metaphor in the 
Educational Partnerships Program and to seek the data that reveal the differences in opportunities 
for students in the Star Schools Program. In both cases, program developers valued what they 
saw as unusual "insights." 

Even when not fiilly used, the model creates a positive environment for evaluation use. 
When we put together the program theories, for example, we share them with key audiences. The 
sharing opens a dialogue in which program staff can say, "That isn't what I think. This is what I 
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think about why the activities I'm putting in place will solve the problem." The explicati(Hi helps 
program developers in their reflections about their program. Their reactions help the evaluators 
refine the evaluation design in order to gather more appropriate information. The interactions, 
then, build a basis for evaluation use. 

In sum, the evaluation approach focuses on the values and assumptions ttiat program and 
project developers use in designing how they will address a problem. Because many of the 
assumptions are implicit, the approach is immediately useful. Further, the approach recognizes 
that project developers may hold different program theories from one another and from the 
program developers. By making the differences clear, the evaluators decrease competition among 
developers, which may lead to greater collaboration. The clarity also enables program developers 
to refine their announcements and/or selection criteria and decide to provide technical assistance. 
The deconstmctive mode advances attention to differential effects of a program 
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